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INTRODUCTION 

The  Product  Development  division  of  the  Navy  Advancement  Center  (NAC) 
designed  a  pilot  product  for  future  Electronic  Training  Manual  (ETM)  development.  A 
print-based  training  manual  (TRAMAN)  and  nonresident  training  course  (NRTC)  were 
converted  into  an  Interactive  ETM.  The  TRAMAN  was  converted  into  a  Standard 
Generalized  Markup  Language  (SGML)  format,  and  the  NRTC  was  converted  into 
interactive  tests  with  remediation. 

PURPOSE  OF  THE  EVALUATION 

The  purpose  of  the  evaluation  was  to  (1)  measure  the  technical  accuracy  of  the  product, 
(2)  measure  the  effectiveness  of  the  electronic  training  environment,  and  (3)  gather 
information  from  the  target  audience  about  the  product.  During  the  evaluation,  the 
following  quality  indicators  were  assessed:  (1)  completeness,  (2)  technical  accuracy,  (3) 
instructional  soundness,  (4)  functionality,  (5)  user  friendliness,  and  (6)  aesthetic 
appearance. 

COURSE  DESCRIPTION 

The  ABE  3  &  2  Interactive  ETM  is  an  electronic  version  of  the  ABE  3  &  2  TRAMAN 
and  NRTC.  The  ETM  is  displayed  with  the  DynaText  browser.  The  browser  has 
keyword  searching,  note-taking,  and  printing  capabilities.  The  student  can  search  the 
ETM  content  for  key  words,  take  notes  while  studying,  and  print  topics  of  interest  or  the 
entire  ETM.  Interactive  testing  with  remediation,  color-enhanced  graphics,  animations, 
audio,  and  hyperlinks  complement  the  ETM.  Authorware  Professional  interactive 
multimedia  software  package  was  used  to  develop  the  interactive  testing. 

The  Interactive  ETM  is  a  supplement  to  the  print-based  TRAMAN  and 
NRTC.  It  contains  the  same  basic  technical  information. 

SCOPE  OF  THE  EVALUATION 

The  evaluation  was  an  alpha  test  that  included  the  interactive  and  printed 

ETM. 


ALPHA  TEST 
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The  alpha  test  included  students  participating  in  one-to-one  and  small  group  evaluations, 
subject  matter  experts  (SMEs),  instructional  systems  specialists  (ISSs),  and  editor?writer 
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evaluations.  This  test  was  designed  to  obtain  feedback  about  the  course. 

PARTICIPANTS 

The  alpha  test  involved  13  students,  5  observers/interviewers,  4  subject  matter  experts 
(SMEs),  2  instructional  systems  specialists  (ISSs),  and  2  writer?editors. 

Students 

Students  participating  in  the  evaluation  were  from  the  student  population  at  the  Naval 
Air  Technical  Training  Center  (NATTC)  at  Naval  Air  Station  (NAS)  Pensacola,  Florida. 
There  were  two  groups  of  students:  (1)  one-to-one  evaluation  participants,  and  (2)  small 
group  evaluation  participants. 

For  the  one-to-one  evaluations  selected  students  were  familiar  with  the  subject  and  had  a 
strong  background  in  computers.  Students  from  the  upper  percentage  ranges  in  aptitude 
and  subject  content  are  often  more  likely  to  point  out  and  analyze  weaknesses  in  the 
student  instructions  and  course  materials. 

For  the  small  group  evaluation  selected  students  had  a  broad  range  of  knowledge  about 
the  subject  and  had  diverse  backgrounds  in  computers.  The  following  criteria  were  used 
to  select  the  students:  (1)  aptitude,  (2)  prior  subject  knowledge,  (3)  computer  skills,  (4) 
attitude,  and  (5)  general  background  experience.  Participants  represented  the  students  in 
the  ABE  3  &  2  training  environment  as  much  as  possible. 

Observers/Interviewers 

Observers/interviewers  were  Saufley  Field  NETPDTC,  Pensacola,  Florida  ISSs. 
Observers/interviewers  described  events  encountered  while  the  students  evaluated  the 
course,  recorded  the  students'  comments,  and  interviewed  the  students  after  they 
completed  the  course.  During  the  one?to?one  evaluations,  one  observer/interviewer  was 
assigned  to  each  student.  During  the  small  group  evaluation,  one  observer/interviewer 
was  assigned  to  every  two  or  three  students. 

SMEs 

SMEs  were  senior  ABE  Sailors  from  the  NAC  and  NATTC.  They  evaluated  the  course 
for  completeness  and  technical  accuracy.  The  SMEs  completed  a  “subject  matter  expert 
survey”  and  prepared  a  “technical  deficiency  report.”  The  SMEs  were  in  a  unique 
position  because  they  have  taken  the  paper-based  course  before  and  furnished  direct 
comparisons  between  the  paper-based  and  ETM  products. 

ISSs 

ISSs  evaluated  the  course  for  instructional  soundness  and  furnished  their  comments  in 
an  “instructional  soundness  evaluation  report.” 

Writer-Editors 
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The  writer?editors  evaluated  the  printed  ETM  by  comparing  it  to  the  original 
print-based  TRAMAN.  They  conducted  a  word-by-word  comparison  of  all  the  text. 
They  ensured  all  the  original  content  appeared  in  the  electronic  ETM.  They  ensured  all 
the  formatting  styles  were  correct.  They  examined  the  figures  and  tables  for 
completeness  and  accuracy.  They  reviewed  the  progress  test  questions  and  answers  to 
ensure  they  complied  with  NAC  examination  standards. 

LOCATION  AND  LOGISTICS 

The  one-to-one  and  small  group  evaluations  were  conducted  for  1  week  each  at 
NETPDTC  in  the  Learning  Resource  Center  (LRC). 
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